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Abstract 

Let Y = G/L he a. flag manifold for a reductive G and K a maximal compact 
subgroup of G. We define here an equivariant differential operator on G/LCiK playing 
the role of an equivariant Dolbeault Laplacian for the complex manifold G/L, using a 
distribution transverse to the fibers of G/LCiK G/L and satisfying the Hormander 
condition. We prove here that this operator is not maximal hypoelliptic. 

Introdution 

There are two challenging problems in representation theory of Lie groups. The first 
one is to classify unitary representations for large classes of Lie groups. Connected 
nilpotent Lie groups form such a class, and Kirillov established, for any connected 
nilpotent Lie group, a bijective correspondance between the set of coadjoint orbits 
and the set of (equivalence classes of) unitary irreducible representations of the group. 
This approach lead to the second problem : to realize unitary representations geo- 
metrically. These two problems are still open for reductive groups, but the technique 
of coadjoint orbits is a constant source of inspiration. For reductive groups there 
are three kind of orbits |VogOO| : the hyperbolic orbits, the elliptic orbits, and the 
nilpotent orbits. The hyperbolic orbits lead to the theory of parabolic induction 
and Knapp-Stein intertwining operators. This is appropriate to construct unitary 
representations that are weakly contained in the regular representation. The elliptic 
orbits are related with the theory of cohomological induction and the geometry of 
flag manifolds. The study of nilpotent orbits lead to the theory of unipotent repre- 
sentations. We are concerned here with the geometry of flag manifolds and we use 
the theory of coadjoint orbits for nilpotent Lie groups to handle regularity problems 
of differential operators on flag manifolds. 

Let G be a reductive Lie group and y be a flag manifold for G. The G-space Y 
is a complex manifold with an equivariant complex structure, and is a homogeneous 
space of the form G/L, where the Lie subgroup L is reductive but don't need to 
be compact. A representation x of L is chosen, and the usual Dolbeault complex 
is twisted by x- The smooth cohomology H*{d^) of this complex is proved by H. 
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W. Wong |Won95j to be a Frechet representation of G whose underlying Harish- 
Chandra module is isomorphic to the cohomologically induced representation R{x). 
The proof of H. W. Wong uses the double fibration G/L <— G/LCiK — > G/K, where 
the group is a maximal compact subgroup of G. One conjecture that if x is a 
unipotent unitary representation of L, whatever it means, then the representation 
H*(d^) is unitarizable. However, as a Frechet space it can not carry a unitary 
structure. In the best case, when L is compact, one choose a G-invariant hermitian 
metric on Y and then consider two objects : the Hilbert space of square integrable 
sections of the twisted Dolbeault complex, and the Dolbeault laplacian = d^d^ + 
9*9^. This differential operator is elliptic and is a selfadjoint operator on the Hilbert 
space. Its L^-kernel is then proved to be a unitary representation that infinitesimally 
isomorphic to the Frechet representation. Such representations are sums of discete 
series |AS77| . [CM82j . In the general case, necessary to find other representations, 
the flag manifold does not carry any G-invariant hermitian metric. A positive metric 
is then defined in [RSW83] to define the Hilbert space, and I proved in full generality 
|Pru06j that this Hilbert space is a continuous G-module. The proof again uses the 
double fibration considered by Wong. To replace the G-invariant selfadjoint operator, 
an invariant non-positive form on Y is defined |RSW83] . |BKZ92| . It is used to define 
the adjoint S^^nv the harmonic space ker9^ n ker9^. The point is that the 
invariant operator d^d*^^^^-^ + d*^ i^-^d^ does not satisfy any regularity condition such 
as ellipticity and can not be used. 

We propose here a new invariant operator, defined via the fibration ttl: G/LD 
K G/L and study its regularity properties as an operator on G/L n K. We 
first define a distribution E transverse to the fibers that satisfies the Hormander's 
condition. It is used to pullback the Dolbeault operator also denoted by d. The 
manifold G/LV\K has a G- invariant positive metric defined by the Killing form, and 
we can use it to define the formal adjoint d of the pullback of the Dolbeault operator. 
We then define □ = 95 +9 9. We first show that on sections constant along 
the fibers, this operator equals (up to an operator of lower order) the Hormander 
Laplacian which is known to be maximal hypoelliptic. We next show that on the 
whole space of sections over G/L n K the operator □ is not maximal hypoelliptic. 
To prove this we provide the tangent space of G/L r\ K with a nilpotent algebra 
structure, canonically associated to the fibration til, and find a non trivial irreducible 
representation a of the associated connected nilpotent Lie group such that the image 
by a of the i?-symbol of □ is not injective on the space of smooth vectors of a. 
Actually this turns out to be the case for many representations. 

The representation x would have been of interest for the (more delicate) questions 
of positivity for instance but does not come into questions of regularity ; we then use 
the usual Dolbeault complex. 

During the preparation of this article I benefited of many helpfull discussions with 
L. Barchini, P. Julg, J.M. Lescure, B. Nourrigat, R. Ponge and A. Valette. 
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1 The Dolbeault Laplacian 



1.1 Definition 

Let Y = G/L he a flag manifold for a reductive Lie group G. This means that Y is 
an open orbit in a flag manifold G'^/Q, where G'^ is the complexified Lie group of G 
and Q is a parabolic subgroup of G'^ ; we also require that Y admits a G-invariant 
measure. We note go the Lie algebra of G, and g its complexification and use the 
same convention with other real and complex Lie algebras and spaces. Then Y has 
an equivariant complex structure. Choices of a maximal compact subgroup K of G 
and of a base point yo can be made such that the reductive group L = StabG(?/o) 
is the centralizer of a compact torus with Lie algebra tf, G floj -C''^ is the Levi part of 
Q and K/L f] K is a maximal compact complex submanifold of Y. The parabolic 
algebra q has a decompostion q = 1 © u, and q = [ © u © u with X i-^ X is the 
conjugaison associated to the real form G of G'^. The space u is L-isomorphic to the 
antiholomorphic tangent space Te'^G/L. Note that the connected reductive subgroup 
L need not to be compact, so that Y does not have a G-invariant Riemannian metric 
in general. 

The manifold Y has a G-invariant complex structure : this means that the De 
Rham operator d writes d = d+B, where d : AP'^TYc AP+^'^TYc and d : AP^^TYc 
;\P^i+^XYc arc G-cquivariant operators. The restriction to A^'*TY = A*u of the 
operator d is called the Dolbeault operator . The manifold Z = G/Lf] K fibers over 
Y and the group G acts on it properly. It then admits a G-invariant Riemannian 
metric. We define the horizontal space at a point z G to be the orthocomplement 
Ez of the space tangent at z to the fiber trough z. 

We then have a connexion E on the fibration ttl which enables to pullback the 
Dolbeault operator. 

Proposition 1. Let Y he a complex manifold with G-invariant complex structure 
and n: Z ^ Y an equivariant fibration, with fiber F. We suppose that the exact 
sequence 

TF^TZ ^ Tr*TY 

has an equivariant splitting. Letp^'^ be the transposed map p* of this splitting followed 
by the projection to the (pullback of the) antiholomorphic tangent space 7r*T^'^Y. 
Then there exists a unique operator d' on Z satisfying the following conditions. 

d'{Tr*u) = Tr*{dij) (1) 
d'{f7r*u;) = pl'^df A (7r*a;) + f7r*{du) (2) 

The operator d will be denoted d when no confusion arises. 
Proof. We have to check that, for any / G G°°{Z), any g G G°°{Y) non zero, and 
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any uj G T{Y, ^*T^^^Y), we have d' {f-K*giT*{g-'^uj)) = d'{fiT*u). Now, 



= p'i'Hdf)7T*g7T*{g~') An*iu;) + fp^,''d{7r*g)7r*{g-') A7r*{uj) 

+ f7r*g7r*d{g-')7r*uj + f7r*ig)7r*{g-')7r*idu) 
= pO'^df A {7r*co) + f7r*{du) + f7T*{g-^dg + gdig-')) A to 
= d' if Tr*u;). 



□ 



The action of G on G/L Ci K is proper. In particular, this G-spacc admits a G- 
invariant Riemannian metric. As usual the choice of such a metric enables to define a 
bilinear pairing ( , ) between the space of forms with compact supports and the space 
of forms. The *-operator is then given by (a,/3)(ivol = a A We then define the 
adjoint of the pullbacked Dolbeault operator (on homogeneous forms) by 



This operator is G-equivariant by construction. The question is : can we build an 
algebra of pseudodifferential operators on which the Dolbeault Laplacian admits a 
parametrix ? The result we prove here gives a negative answer to that question. 

1.2 Structure of the transerve subbundle 

The Dolbeault Laplacian is clearly not elliptic. To study more involved regularity 
properties of this operator, we will need detailed information on the bundle E. We 
now investigate the structure of this bundle. 

Definition 2. A subbundle E of the tangent space TZ is a 2-step bracket gener- 
ating subbundle if for any point p ^ Z, the space [X,Y](p) mod Ep, with X and 
Y running over sections of E,is the whole space TpZ/Ep. In particular the bundle 
homomorphism 



induced by the barcket [,] of vectors fields, is onto. We say that E satisfies the 
Hormander condition at order 2. 

Lemma 3. The subbundle E of the tangent bundle satisfies the Hormander condition 
at order 2. 

Proof. Without lost of generality we may assume that g is simple. It is enough to 
prove that s = u © u + [u, u] is a non zero ideal of g. As q is a parabolic subalgebra, 
u and u are sums of root spaces. Moreover g = I © u © u, so we get 



5*a; = (-l)H(*5*)a;. 



It remains to define the Dolbeault Laplacian by 



a = dd* + d*d. 




(3) 



[u, u] = ([u, u] n [) © ([u, u] n u) © ([u, u] n u) . 
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Using this one checks that [u©u, [u, u]] C s. Let X,X' € [u, u]. We write X = 
Xi + Xu + Xu thanks to the preceding equation, and X' = [X'^,XQ. This gives : 
[X, X'] = [[Xi,X^],XL] + [x;, [Xi,XL]] + X" with X" G 5. So [X, X'] G 5 and s is a 
subalgebra of q. It is also clearly stable by [. □ 

We now state a more precise result when G is the group U (p, q) and L = U{pi) x 
U{p2, q) with pi+p2= p. 

Lemma 4. There exists a sequence F = (71, . . . , 7^) of roots in A([np), such that, for 
any a G A(u) there exists at most one 1 <i <r and /3 G A(u) such that azizji = (5. 
Moreover, such an a exists for all compact roots in A(u) or it exists for all non 
compact roots in A(u). 

Proof. The roots in A are Cj — Cj and 

A([np) = {ci - Cj ;pi < i < p < i < p + q] 
A(u) = {ci - Cj ;l < i < pi < j < p + q] . 

Set r = min{pi , q\ (the real rank of the noncompact semi simple part of [) and 
let r = (7j) be any set of strongly orthogonal roots. For exemple, one may take 
7i = Cpi+i — Bp+g-j. The result follows easily. In fact, if a = — Cj G A(u) then the 
only (3 = Ck — ei that may work are those with k = i or I = j, and only one of them 
can lies in F. □ 

Remark 5. This lemma is also easily seen to be true when G is any real rank 1 
group. 



1.3 Statement of the main result 

Let us precise now the regularity property of differential operators we want to in- 
vestigate. Let Xi, . . . ,Xk be vector fields on a neighborhood ^ of a point xq G M", 
and let be the subspace of generated by the vectors Xj(rco). We also assume 
that vectors [Xi, Xj]{xo) mod (-E^o) generate the vector space W^/Exq. The space 
of operators of order less than m is the space of operator P that can be written in 
the form 

P = a„ (x)X" , X" = Xf ^ • • • Xf , (4) 

\a\<m 

where the coefficient aa{x) are smooth functions of the variable x on V. 

Definition 6. \HN85\I A differential operator P of order m is maximal hypoelliptic at 
xq, if there exists a neighborhood V of xq and a constant G so that for allu G G'^{V), 

\\X''u\\l2<G{\\u\\l2 + \\Pu\\l2). 

\a\<.m 
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Maximal hypoellipticity of an operator P implies that P is hypoelliptic, i.e. 

Pu smooth ^ u smooth . 

The principal £'-symbol is by definition p = Yl,\a\=m^a{x)£,'^ ■ We will use the sign 
"~" to say that two operators have the same principal E-syTobol. The following 
result is well known (see [ HN85j ). 

Proposition 7. The Hormander Laplacian Xf is maximal hypoelliptic. 

We choose here the metric given by the Killing form B. More precisely, the metric 
is defined at the origin by 

{X,Y) = -B{X,e{Y)). 

This form is definite positive on g and is t-invariant. The tangent spaces TeY ~ u©u 
and TgZ ~ u © u © (t H p) are provided with this hermitian metric. 

Theorem 8. Let G = U(p, g) and L = U(pi) x \J{p2,q), with pi + p2 = P- The 

Laplacian □ is not maximal hypoelliptic at the origin eL n K. 

One may conjecture this this result is true for any semisimple Lie group and flag 
manifold. The exposition of the proof is intended to make clear that only the lemma 
m as to be generalized. So let G be a reductive Lie group with a compact Cartan 
subalgebra, and G/L be a flag manifold for G. This assumption on the Cartan 
subalgebra makes less technical the computation of the principal E'-symbol , but we 
should proceed without it. 

The next section is devoted to the proof of the theorem [51 To prepare the proof 
we compute here the local expression of the principal E-syTobol of this operator. The 
Cartan subalgebra being compact, we may suppose that 

to C [q n 6o C [q C So • 

Let A be the root system of the pair (g, t) All roots of the root system A(g, t) being 
compact or non compact, it makes sens to define A(u H 6) and A(u R p) and so on. 
We choose a system A"'"(g,t) of positive roots such that A(u) C A+(g,t). As the 
Killing form is non-degenerate there exists for any a G A a vector Ha S t so that for 
all H £i, a{H) = B{H,Ha). 

Lemma 9. There exists an orthonormal basis {Ea)a<^A of root vectors satisfying 

[Ea,E^a]=Ha (5a) 
[Ea,Efi] = 7V„,/3^a+/3 with N^^p = Oifa + p(^A (5b) 
Na,p = -N.a,~p ■ (5c) 

Proof. According to [ Hel62t theorem 5.5] there exists a basis {E'^) satisfying equations 
(P. The relation implies that B{E'a,E'_J = 1. Moreover B{E'a,EL) = if 
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a + /? 7^ 0, and \\E'^\\ > 0, so it follows that —9{E'^) = c-aE'_a, with CaC-a = 1- We 
now define Ea = XaE'^ where XaX-a = 1 and = — Cq,. We then get 

—9{Ea) = XaC-aE'_a = —E-a and (6) 
= x„x_«K, = Ha . (7) 

So that {Ea,Ea) = B{Ea, E-a) = 1 and the basis (Ea) is now orthonormal. Using 
equations (0) and ([7]) is easy to check that the basis (Ea) again satifies the equations 
©. □ 



We set Za = Ea and 



—E-a if ci is compact, 
£"0 if a is non compact. 



This notation is concording with the complex structure. Let us now define the real 
vectors Xj and Y^. 

I i — 

Xj = — [Z^ + Z^) , = - — [Zy - Zj) . 

The sytem (^7, 5^7)'ygA+\A+([ns) is an orthonormal basis oiT^Z and (^7, 5^7)7gA{u) is 
an orthonormal basis of E^ ~ T^Y. Moreover if J denotes the complex multiplication 
operator, one has Y^ = JX^, for 7 G A(u) (and 7 G A([ n p) when L/L n is a 
hermitian symertic space). We also have 

1 i 

Xa = —i={Ea — E-a) Ya = -/={Ea + E-a) if " is compact, (8a) 

V 2 V 2 

1 i 

Xp = —={Ep + -E-/3) Yp = -/={Ep — E-p) if /? is non compact. (8b) 

\/2 v 2 

Proposition 10. For a G A(u H 6) and (3 G A(u H p) we have 

[Xa,Xi3] = -j=(^Nai3Xa+i3 + Na-i3X\a-/3\J (9a) 



[Xa, Yp] = l=(^Na,(^Ya+(^ - e(a - P)Na,-(^Y^a-p\ 



a-l3\ 



(9b) 
(9c) 



[Ya,Xp] = —i^Na,pYa+p + e(a - P)Na,-(3Yl 

[Ya, Yfs] = -l=(^Na,f3Xa+f3 " Na,-f3X\a-f3\) (9d) 

The vectors involving roots of the form a + (3 lie in E^. The vectors involving roots 
of the form a — (3 may lie in F^, hut don't need to. Other brackets of base vectors lie 
in Ee- 

To prove this proposition one just computes using equations ()5b|5cp and the fact 
that if a G A(u H 6) is compact and jS G A(u H p) is non compact then a ± /3 either 
is a non compact root or is not a root. 



7 



Let be the exterior multiplication by Z^. Then the Dolbeault operator has the 
following principal E'-symbol. 

7eA(u) 

where is here the left invariant vector field generated by Z^. Let be the interior 
multiplication by with respect to the chosen metric. Then 

5* ~ — i^Z^ . 

7eA(u) 

According to the previous notations these equations become 



9^ E ^(^7-^^7), 
7GA{u) ^ 

7eA(u) 

It now remains to compute. 

77' 7' 7 



Let us write the diagonal terms separetely. 



□ --^ E (67^7 + ^67) (^7 + ^7 ) 



7GA(u) 



1 



77^7' 



x^x^/ + y^y^/) + i {x^Yy + Y^Xy 

XyX^ + yyy^) + i {YyX^ + Xyy^) 



We have e-yiy + iye-y = 5yy (Kronecker' symbol). 



E (^7 + ^. 



7GA(u) 



1 



e^iy \ {[X^, Xy] + [Y^, Yy]) + i ([X^, yy] + [Y^, Xy]) 



77^7' 

Using proposition \W\ one gets 



[X^,Xy] + [Y^,Yy] = ^/2X,,_^X| 
[X^,yy] + [Y^,Xy] = -V2N^^_pY\ 



\a-f3\ 



and 



\a-f3\ 
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if 7 = a is compact and 7' = /? is non compact. One has similar relations when 7 = /3 
is non compact and 7' = a is compact. Other brackets are horizontal and they don't 
appear in the principal i?-symbol. This gives 



7eA(inp) 

+ i (5^*^^a,/3 (eai/3 + epioS) y-y] (10) 



7gA(u) 7eA(inp) 



where the smns ^* are over a G A(u n 6), /? G A(u n p) and \a — (5\ = 7. The local 
formula (llOh will be used later in the proof of theorem [8l 

This formula is already usefull for functions. In fact, the terms of classical order 
1 vanish on functions, so □ is maximally hypoelliptic when restricted to functions 
because it has the same principal S-symbol as the Hormander Laplacian (up to a 
constant). 



2 The Rockland condition 
2.1 Hypoellipticity criterion 

For the proof of the theorem [8] we use techniques of Folland and Stein |FS74j . We 
now provide the tangent space TgZ with a nilpotent Lie algebra structure no- This 
structure is given by the brackets [, ]o, and the identification of TZ/E with F. The 
Lie brackets [, ] is then given as follows. Compare with proposition I lOi 

Definition 11. For a € A(u n i) and [3 e A(u n p) we have 

ix^,xp\ = -L(iv;_^X|„_^|) (iia) 

lX^,Yp\ = J-{^-e{a- ma,-py\a-p) (lib) 



1 

3^ 



[y„,X^] = — e(a - f3)K,-pY\^-p\] (He) 



{Y^,Yp\ = -^(- iV;-,3^|«-/3|), (lid) 



where N'^ _^ = N^-is if a — (3 & A([n p) and otherwise. All other brackets of base 
vectors are defined to be 0. 

Let P be a differential operator on an open set of M" as in the first part, with 
principal i?-symbol p. We say that P satisfies the Rockland condition if for any 
unitary irreducible non trivial representation vr of the simply connected nilpotent Lie 
group N = exp(no), the operator 7r{p) is injective on the space of smooth vectors of 
vr. The sympbol p is seen here as an element of the enveloping algebra U{n) of n. 

Theorem 12. \HN85\I The following are equivalent 
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1. P has a parametrix in the E-pseudodifferential calculus, 

2. P satisfies to the Rockland condition, 

3. P is maximal hypoelliptic. 

Let be a nilpotent Lie group with Lie algebra no- Then acts on tXq by 
the coadjoint representation. Kirillov defined a one-to-one correspondance between 
coadjoint orbits and (equivalence classes of) irreducible unitary representations of 
the group N constructed in three steps as follows. 

Lemma 13. Let I he a form on no and Bi: {X,Y) i-^ /([X, y]). Then there exists 
an isotropic suhalgehra f)o of x\.q for Bi such that codimf)o = ^ranki?^. 

Then exp(iZ) is a one dimensionnal representation of the nilpotent group H = 
exp(f)o). 

Lemma 14. The induced representation Ind^e*' is irreducible and its class only 
depends on the coadjoint orbit of I . 

There is also a converse statement. 

Lemma 15. All the irreducible representations of N arise in this way exactly once. 

We will need to recognize induced representations realized on M". Let vr be a 
representation of the nilpotent Lie algebra no on 5(M"). We suppose that, for any 
X S no , the operator t^{X) has the form 

where Pk{-]X) and Q{-]X) are polynomials on depending linear ily on X. We 
also assume that the linear forms 5,k{X) = Pfc(0;X) are linearily independent. Let I 
be the linear form on no defined by 1{X) = (5(0; X) and {)o = nker^^. 

Proposition 16. J HNS 51 Proposition 1.6.1] Under the above assumptions, the sub- 
space f)o is a subalgebra o/ no, isotropic for Bi. Moreover, the representation vr is 
unitarily equivalent to Ind^e*'. 

2.2 Proof of the main theorem 

Here we prove that the evaluation of the Dolbeault laplacian has a kernel of positive 
dimension on many representations under conditions on root systems. The choice of 
these representations and the proofs of the root systems conditions are made for the 
groups G = U(p, q) and L = U(pi) x \J{p2,q), with pi+p2 = P- One can expect that 
this can be done in full generality. 

We now have to find unitary irreducible representations of the connected nilpotent 
Lie group H and to realize them on L^(R"). This will lead to a partial differential 
equation on M". In other words the linear form on no that gives the representation 
of A^, has to be taken such that the obtained partial differential equation (can be 
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solved and) has a non zero solution space. Let / € rig be a linear form on no with 
coordinates (C-yiV-y) ™ the dual basis of {X^,Yy). Let vr; be the representation of N 
associated to the coadjoint orbit of /. 

Using definitonllll one find that the form Bi: (X,Y) 1{[X,Y]) as a martix of 
the form 




with A 



V2 



C\a-I3\ - P)r]\a-I3\ 



a,f3 



We make the following assumption on /. 
(H) A has a maximal rank. 

If hypothesis (H) is true then 

(H') either po or [q n po © (u © u) n to is a maximal abelian subalgebra of no. 

This means that the hypothesis (H) is more an hypothesis on the pair (G, Q) than 
on the linear form I. Let us assume hypothesis (H). Let f)o be the abelian subalgebra 
of no such that 

f)o = Po if dim po = max{ dim po ; dim lo n po © (u © u) n 60 } , 
and f)o = [q n Po © (u © u) n 60 otherwise. Then 

codimfio = -rankS; , 
and Pio is an isotropic subspace for Bi. 

Lemma 17. Let G = V{p,q) and L = U(pi) x V{p2,q)- There exists a linear form 
I such that hypothesis (H) is satisfied. 

Proof. Take I be non zero on root vectors correszponding to a set of strongly orthog- 
onal roots in A+(l n p) such as in the proof of lemma HI and elsewhere. Then A is 
" diagonal" with no zero on the diagonal, by lemmas I3I4I □ 



First case. Let us begin with the case [)o = po- Let s = d\m.icK/L r\ K = dimu n t. 
Then vr; = Ind^e^^ is a unitary irreducible representation of N on L^(no/f)o) that 
can be seen as a representation on L^(M^'^). We note {xa,ya)a&/\{vC\t) the canonical 
basis of "M?^. Thanks to proposition 1161 we have 



d 

dXn 



d 



+ irja 



T^liXp) = i 
7r/(ya) = i 



E 

a 

E 



dya 

P)V\a-f3\ya) 



(e(a - P)r]la-/3\Xa + C\a-I3\ya) 

= iC'y, 7ri{Yy) = irj^ . 
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To make the computation more easy we also suppose that 

= Va = ^(3 = V/3 = ■ 



(12) 



This is a priori not true in general that any orbits admits a form of this kind, but this 
is enough, to prove the theorem, to find such forms such that iri{\Il) is not injective. 
Then, the operator tti (□) has the following form. 



dxl^ dyl """^^"^^"^ 



(13) 



N' 



where Va is the positive real number such that = ^* "^^ i^\a-/3\ + ^ia-/3|)' ^^"^ 

iC\a-P\ + iV\a-P\)eaif3 - (C|a-/3| " iV\a-P\)ifSea 



M, 



a,l3 



V2 



is an endomorphism of A*u and the sum J2* is over the set of roots (3 G A(u fl p) 
such that a- P € A(l n p). 

and Ma = T.*Ma,8- We have to find 



Let 



a2 , 92 



eigenvalues of and Mq of opposite signs and the same absolute value. 

Making the change of variables 



1 



the operator Da becomes — ^ 



. It is — ^ times the Hermite 



{xl + yl) 

operator of dimension 2. Its eigenvalues are then — fcr^, with k G N*. As the operators 
Da differentiate on different variables, we see that the eigenvalues of Da are 
— 'Y^a ^ai"a > with ka G N* . Wc also note that the eigenfunctions of the Hermite 

|j:||2 

operator are of the form P{x)e 2 where P is a polynomial in a; = [xi, . . . ,X2s)- 
So they are in the Schwarz space, so are smooth vectors of the representation tt;. 

Let us now show that ± J2a eigenvalue of J2a -^a- ^^^^ show that 

Tq is an eigenvalue of Ma- Let A(u n t) = {ai, . . . , Ug} and v = Z^-^ A 
13 ^ P', then Ma,/3Ma,p'{v) = Ma,/3' Ma,p{v) = and moreover 

Ar'2 



AZa,. If 



It follows that 

M^{v) = j2*Kp('^) = ^l^- 

So the vector ivaV + Mq,v is an eigenvector for M^ with eigenvalue irQ. 

Proposition 18. Let k < s and {n; • • • ;ik} C {1; • • • ; s}. T/ten nf=i ^^at^v does 
not depend on the order of the ii . 
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This proposition is easily checked by induction on k. We now define by induction, 
for A; < s, the vectors Vk hy vq = v and 

The preceding proposition shows that if Vk-i is an eigenvector for Mq^, / < k, with 
eigenvalue r^i, then is again an eigenvector for M^i, I < k, with eigenvalue r^^. 

Lemma 19. Let G = U(p, g) and L = U(pi) x U(p2)<?)- There exists a linear form 
I on no satisfying hypothesis (H), and such that Vk is an eigenvector for Ma,., with 
eigenvalue r^j. . 

Proof. Defining I has in the proof of lemma [17] again works. □ 

Hence Vg is a simultaneous eigenvector for all M^'s, with respective eigenvalue 
ra- So Vg is an eigenvector for Ma with eigenvalue r^. We end this first case 
f) = po remarking that the constructed eigenvector lies in A'^u, and this means that 
□ is not maximally hypoelliptic on degree s = dime K/L n K. 

Second case. Let us now assume that t = dimu H p < s. Switching the role played 
in the first case by the a's and the /3's, one similiraly proves that □ is not maximal 
hypoelliptic on A*u. Using the duality 

A : A* u A^u ^ A^'^^'u , 

one shows that □ is not maximal hypoelliptic on degree s in the second case too. 

Finally, we have shown that □ is never maximal hypoellitpic on degree s and 
on the complementary degree t. It would be remarkable if these degrees are the 
only one where this phenomena arises. It is obviously the case for G = U(2, 2) and 
L = U(l) X U(l,2) for instance, because any coadjoint orbit admits a linear form 
satisfying equation p^ . 
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